BACKGROUND OF THE INVENTION 



1. Field of the invention 

This invention relates in general to video processing 
and in particular to a method and device for providing instant 
replay of video. 

2. Description of the prior art 

Instant replay permits a viewer to rewatch a portion of 
a video program for closer study, e.g. to rewatch a play in a 
sporting event. In U.S. Patent No. 5,371,551 an instant replay 
device is provided with a circular memory for recording video for 
later replay. Each frame of video is simply stored in a memory 
at an address equally spaced from the next frame, which means the 
memory in U.S. Patent No. 5,371,551 does not take advantage of 
current MPEG standards which permit variable length encoding of 
frames . 

In addition current MPEG- 2 decoders receive the video 
information in the form of a compressed bit stream. The order of 
the pictures (e.g. I,P,B,B) in the compressed bit stream is 
different than the decompressed order in which the video is 
displayed (e.g. I,B,B,P) . This reordering is necessary to ensure 
that all of the anchor frames needed to reconstruct the current 
frame have been received by the decoder first. The MPEG syntax 
does not contain any element which correlates the order of the 
pictures in the compressed domain to the order of the pictures in 
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the decompressed domain. 

For operation of instant replay some correlation would 
be advantageous because a user typically views the decompressed 
video and then selects a portion of this video to be instantly 
replayed. This portion is then retrieved from memory and 
redisplayed. Since there is no correlation in MPEG between the 
decompressed video that is displayed and the compressed video, 
the storage device must store the decompressed video in order to 
begin replaying the video at the appropriate point in the video 
stream. This requires large costly memories. 

SUMMARY OP THE INVENTION 

Accordingly it is an object of this invention to 
provide instant replay for an MPEG encoded video stream, 
including video streams received via live terrestrial 
transmission, by storing the data in the compressed domain and 
providing non-MPEG frame tags for correlation to the decompressed 
domain. 

It is another object of the invention to provide 
instant replay of video without adding substantial cost to the 
video decoder. 

It is a further object of the invention to provide 
instant replay which is not limited to any one profile or level 
of MPEG. 

It is yet another object of the invention to provide 
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instant replay in a system which optimizes the use of the memory 
by utilizing the benefits of variable length encoding. 

Still other objects and advantages of the invention 
will in part be. obvious and will in part be apparent from the 
specification. 

The invention accordingly comprises the several steps 
and the relation of one or more of such steps with respect to 
each of the others, and the apparatus embodying features of 
construction, combinations of elements and arrangement of parts 
which are adapted to effect such steps, all as exemplified in the 
following detailed disclosure, and the scope of the invention 
will be indicated in the claims. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a fuller understanding of the invention reference 
is had to the following drawings: 

Fig. 1 shows an example of the components of prior art 
video decoders; 

Fig. 2 shows a block diagram of the FLD, VLD, rate 
buffer, replay memory, rate buffer control and replay controller 
in accordance with the invention; 

Fig. 3A shows the components of the video stream in the 
decompressed display order; 

Fig. 3B shows the components of the video stream in the 
compressed order; 
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Fig. 4 shows a more detailed view of the replay memory 
and replay controller; and 

Fig. 5 shows the elements of a video decoder in 
accordance with the invention. 

DETAILED DESCRIPTION OP THE PREFERRED EMBODIMENTS 

Figure 1 shows the components of present MPEG video 
decoders. The fixed length decoder (FLD) 10 decodes the 
syntactical information in the headers- of the MPEG video stream 
from the PES layer to the Picture layer and provides this 
information to many of the remaining decoder elements. The FLD 
10 also indicates to the rate buffer 12 the beginning of the 
compressed video sequences, pictures and/or GOPs etc. The data 
arrives at the rate buffer 12 at a slower data rate than the data 
rate required for decoding and viewing. By buffering the data in 
the rate buffer 12 the data can be synchronized to the audio 
stream and to time stamps used for buffer management and provided 
to the display at a real time data rate. 

The variable length decoder (VLD) 14 transcodes the 
variable length code words of the compressed video data into run 
length codewords. The run length decoder (RLD) 16 transcodes run 
length code words into quantized coefficient values. The inverse 
quantizer (IQ) 18 maps quantized coefficient values to non- 
quantized coefficient values e.g. discrete cosine transform (DCT) 
coefficient values. The inverse discrete cosine transformer 
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(IDCT) 20 performs an inverse transform operation on the DCT 
coefficients to produce pixel value differences. The motion 
compensation reconstruction subsystem 22 takes the pixel value 
differences and uses motion vectors and the frames (e.g. I and P 
frames) stored in the anchor frame stores 24 and 26 to build the 
decompressed video frames. The I frames are stored in anchor 
frame store 24 and then displayed. As the I frame is displayed, 
the following P frame is reconstructed and stored in the frame 
store 26 and the B frames are decoded by the reconstruction 
subsystem 22 . The decompressed pixel values are displayed by the 
display processor 28. 

As stated above the rate buffer 12 stores the 
compressed video pictures. The display processor 2 8 displays the 
decompressed video pictures. The order in which the decompressed 
pictures are displayed is not the same as the order in which the 
compressed pictures are stored. So, for example, if the 
compressed video stream is I lf P 4 , B 2 ,B 3 the display order 

may be I 1; B 2 ,B 3 , P 4 .... This reordering poses a problem for 
implementing instant replay. The problem is as follows: assume 
instant replay of the last two displayed pictures is requested 
while the P 4 picture is being displayed. In the decompressed 
domain the B 3 and P 4 picture would be redisplayed, since they are 
the last two pictures. In the compressed domain the B 2 and B 3 
pictures would be redisplayed, as these are the last two pictures 
stored in the rate buffer 12. Since, the user is watching the 
decompressed video it is this video (B 3 , P 4 ) that the user would 
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. like replayed, but the rate buffer 12 does not hold the pictures 
in this order (B 2 , B 3 ) . If a request for instant replay is 
received by the video decoder it must determine which picture was 
last displayed and retrieve and redisplay the last X number of 
pictures depending on the length of instant replay requested. 
This means that if the P 4 frame is being displayed (i.e. the 
fourth frame of the decompressed video information) , and only the 
compressed frames are stored, the replay process must correlate 
the decompressed displayed P 4 frame to the position in the rate 
buffer 12 of the compressed P 4 frame (i.e. the second frame of _^ 
the compressed video information) . (This example disregards the 
need for anchor frames as explained below) . 

In addition once the frames are in the decompressed 
domain the frame type information, whether a P, B or I frame is 
being displayed, is lost. The instant . replay decoding process, 
however, needs this "frame type" information as it must start the 
decoding process with an I frame in order to decode the frames in 
accordance with MPEG. Therefore, the nearest I frame to the 
first instant replay frame to be decoded must be found in the 
rate buffer 12 which means frame type information (I,P,B) must 
also be retrieved from the compressed domain. 

Figure 2 shows a video decoder which solves this 
problem. The video picture information is received from the FLD 
10 in the rate buffer write control 32 of the rate buffer control 
30, and in the replay write control and tag insertion 42 of the 
replay memory control 40. The rate buffer write control 32 
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. writes the video picture information and the necessar y head er 
information which dictates the type of picture (I,P,B) into- the 
rate buffer 12 and provides pointer values to the rate buffer 
pointer store 34 which pointer values indicate the starting 
address of the pictures. There is no requirement that the 
pointer values must keep track of "pictures" as in an alternative 
embodiment the pointer values could indicate the beginning of 
sequences, groups of pictures (GOPs) , I frames etc. The rate 
buffer read control 36 receives the pointer values from the rate 
buffer pointer store 34 and reads the picture information from 
the rate buffer 12. 

The invention provides the addition of marker tags to the 
video picture information. The replay write control and tag 
insertion 42 adds tags to each picture , each tag having a unique 
counter value (e.g. 0 to 1799). The range 0-1799 (1800 pictures) 
allows for up to 60 seconds of storage of 30Hz material (1800 
frames of 30 frames/sec material = 60 seconds of replay video) . 
The invention is not limited to counter values (0-1799) as other 
types of tags could be used. 

During decode the FLD 10 detects the start of the pictures 
(the beginning point of the video information for each picture) , 
flagging each start location in the bit stream. The^m arker tags 
are placed into the partially decoded bitstream by the replay 
write contf6r~~and'"tag~* insert ion 4-2 c r ea 'fring ; ja pod 1 ITi ed bit 
stream which is written to both the rate buffer 12 and the replay 
memory 44. The marker tags are stored in the replay pointer 
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-store and mapper 48 and mapped to the pointers. _ The poin ters 
indicate the addresses where the start of the pictures are 
stored. Fig. 4 shows one example of the replay memory 44 and its 
control features. 

5 The rate buffer read control 36 receives the pictures 

along with the marker tags and provides them to the VLD 14. As 
seen in Fig. 5 the tag is carried throughout the decoder to the 
display processor 28. 

If the display processor 28 receives an instant replay 
10 request it provides the marker tag of the presently viewed frame 
to the replay read control 46. The replay read control 46 
provides the marker tag to the replay pointer store and mapper 
3} 48. The pointer store and mapper 48 maps the marker tag to the 
q address in the replay memory 44 of the location of the video 
15 j«| frame having the corresponding marker tag. The replay read 
;U control 46 then rolls back its address to the address of the 

start -of -replay video frame occurring approximately 1800 frames 
;;[] before the marked frame. If the start -of -replay video frame is 
not an I frame the nearest I frame in the bit stream is used and 
20 this frame becomes the new start -of -replay video frame. The 
method in which this I f rame is found is explained below but 
involves the replay read control 46 accessing the replay memory 
location of the start -of -replay frame and checking the header 
information of this frame to see if it is an I anchor frame. If 
25 the start -of -replay frame is not an anchor frame an earlier 
pointer value is requested from the replay pointer store and 
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-mapper 48. For example, in one embodiment the address roll back 
can be determined with reference to stored marker tags. The 
pointer store and mapper 48 stores the information indicating the 
nearest I frame and provides the address of the nearest I frame 
as the start -of -replay video to the replay read control 46. In 
other words, the marker tag itself for each frame references the 
nearest I frame. The replay video is then provided to the VLD 14 
for decoding. One of the benefits of this invention is that the 
replay memory 44 stores the compressed video which means the 
replay memory 44 is relatively small as compared to storage 
required for the decompressed video. In addition, because the 
marker tags reference the address of the start of pictures there 
is no need to store the pictures equally spaced from one another 
to maintain easy referencing. Instead, each frame is stored 
using as little memory as required for that particular variable 
length encoded frame, and direct correlation between the 
compressed and decompressed domains is still maintained. 

The following is a description for detecting the first I 
frame for instant replay. Since the video data is stored in the 
compressed order, all anchor frames needed to decode the current 
frame are stored in memory before the current frame. The anchor 
frame searching schemes described below can be performed by the 
replay pointer store and mapper 48 in conjunction with the replay 
read control 46. 



If the st art -of -replay frame from the instant replav memory is an 
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- 1 frame 

The replay memory controller 40 reads this I frame into the 
video decoder. The replay read control 46 rolls its address 
forward to the next P frame. If reference is made to Fig. 3B # 
5 this means the I 0 frame is read and then the P 3 frame. The P 3 

frame is decoded from the I 0 frame, then all frames after the P 3 
frame are decoded. (See example 1 below) . 

If the start -of -re p lay frame from the instant renlav memory i s a 

10 P frame 

The replay memory controller 40 looks for the preceding I 
Jf) frame in memory. This I frame (e.g. I 0 in Fig. 3b) is used to 
m decode the start -of -replay P frame (e.g. P 3 ) . Then, the memory 
controller 40 reads forward in memory to the anchor frame 
15 % following the start -of- replay P frame. If this anchor frame is 
also a P frame (e.g. P 6 ) , then the preceding I frame is used to 
Y p decode this P frame. Both the start -of -replay P frame and 
tfl following P frame are used to decode the B frames following 
H start -of -replay (e.g. I 0 is used to recover P 3 and P 6 ; P 3 and P 6 
20 are used to recover B 4 , and B 5 ) . See example 2 below. If the 

anchor frame following the start -of -replay P frame is an I frame, 
then the start -of -replay P frame and that I frame are used to 
decode the B frame (s) following the start -of -replay P frame (see 
example 3 below) . 

25 

Tf t-.hft start- of -replay frame from the instant replav memory is a 
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B frame 

The replay controller 40 searches back through the instant 
replay memory 44 until a P frame (here labeled P! for clarity) is 
found. P a is needed to reconstruct the start- of- replay B frame, 
and is retrieved from the instant replay memory. At this point 
the replay controller 40 searches for an I frame. If the P^ 
frame is immediately preceded by an I frame, then it searches no 
further: the I frame and the P a frame thus found are used to 
reconstruct the start -of -replay B frame. If the P a frame is not 
immediately preceded by an I frame, then the preceding P frame, 
along with the Pa frame, are used to reconstruct the start- of - 
replay B frame. However, both P frames must themselves be 
reconstructed from the preceding I frame. Thus, the replay 
controller 40 must find the preceding I frame (skipping over any 
further P frames it may encounter) . Once the I frame is found, 
it is used to decode the two P frames, which in turn, are used to 
decode the start -of -replay B frame. (See examples 4 and 5 for 
clarification) . 

EXAMPLES 
Example 1: 

Assume the start -of -replay frame is found to be I 9 in 
Figure 3b. I 9 is loaded into the video decoder, then P 12 is 
decoded, then B 10 and B n (note: it skips B 7 and B 8 since they 
won't be in the GOP started by I 9 ) • After B 10 and B n# it decodes 
as normal. 
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Example 2 : 

Suppose the start -of -replay frame is P 3 in Figure 3b. 
First the replay read control 46 reads back through the replay 
memory to find the preceding I frame (e.g. I 0 ) - Then the replay 
read control 46 searches forward in memory to the anchor frame 
following P 3 . In this case, the next anchor frame will be P 6 . 
Then the decoder uses I 0 to decode both P 3 and P 6 . Then P 3 and P 6 
are used to decode B 4 and B 5 . After B 4 and B 5 , normal decoding 
proceeds . 

Example 3 : 

Suppose the start -of -replay frame is found to be P 6 in 
Figure 3b. The replay read control 46 first looks for the 
preceding I frame (in this case, I 0 ) . Then it looks for the 
anchor frame following P 6 , (in this case, I 9 ) . The video decoder 
uses I 0 to decode P 6/ then uses P 6 and I 9 to decode B 7 and B 8 . 
Normal decoding proceeds after B 8 

Example 4 : 

Assume the start -of -replay frame is found to be B a in 
Figure 3b. The preceding P frame (in this case P 3 ) is then 
found. Then the replay memory controller searches for the 
previous I frame (in this case I 0 ) . The video decoder uses I 0 to 
decode P 3 . The video decoder uses I 0 and P 3 to decode B a . The 
video decoder decodes normally thereafter. 
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-Example 5: 

Assume the start -of -replay is found to be B 5 in Figure 
3b. The replay memory controller first looks back earlier in 
time to find the first P frame (in this case, P 6 ) . Then it 

5 searches further for an I frame. At one point, it comes across 
P 3 . This indicates that the B frame in question will have to be 
decoded from two P frames. Therefore, P 3 is noted. Looking 
further back into memory, the replay memory controller comes 
across I 0 . The decoder uses I 0 to decode P 3 and P 6 . The decoder 

10 uses P 3 and P 6 to decode B 5 . The decoder proceeds with normal 
decoding after B 5 . 

i;n In an alternative embodiment of the invention the 

□ replay memory 44 and the rate buffer memory 12 can be combined. 
15 □ The pointer address of each frame can be used as the marker tag 
'■»% and passed along the decoding chain. The pointer address can 
then be used to locate in the combined memory the present 
compressed video frame being viewed. Such a scheme would 
i s " eliminate the duplication of the control systems 30 and 40 and 
20 the memories 12 and 44 . 

It should be noted that the invention has been 
described with reference to storing the slice layer of the MPEG 
stream, but it can also be implemented in other layers such as 
the transport layer. In addition although the invention was 
25 described with respect to MPEG1 and MPEG2 , it can be implemented 
in any MPEG video syntax that uses compressed and decompressed 
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-video signals in the decoder. 

A further note, if the video stream includes SMPTE time 
codes then these can be used as the marker tags. 

It will thus be seen that the objects set forth above, 
among those made apparent from the preceding description, are 
efficiently attained and, since certain changes may be made in 
carrying out the above method and in the construction set forth 
without departing from the spirit and scope of the invention, it 
is intended that all matter contained in the above description 
and shown in the accompanying drawings shall be interpreted as 
illustrative and not in a limiting sense. 
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